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Detailed Action 



Drawings 

Objections 

1 . The drawings are objected to. 

a. To ensure clarity and proper reproduction, the Applicant is advised to replace the 
handwritten labels and captions of Figs. 4, 9A-9B, 1 1 and 12 with printed ones. 

b. The hand-drawn flowchart depicted in Fig. 12 contains spurious lines and visible erasures 
that render the text and arrows blurry or otherwise unclear. It is suggested that the current 
flowchart be replaced with one that has been more competently drafted. 

A proposed drawing correction or corrected drawings are required in reply to the Office action to avoid 
abandonment of the application. The objection to the drawings will not be held in abeyance. 



Claims 

Objections 

2. Claim 29 is objected to because of the following informality. Notice on lines 10-1 1 of Claim 29 that the 
size of the first and second windows is denoted as 0x0. This notation should be replaced with cox co. The latter 
notation would be consistent with the specification and provide an implied meaning for the parameter co in NCC . 

e> 

Appropriate correction is required. 
Rejections Under 35 U.S.C. § 1 12(2) 

3. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 



The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject 
matter which the applicant regards as his invention. 
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4. Claims 2-3 and 19-20 are rejected under 35 U.S.C. 1 12, second paragraph, as being indefinite for failing to 
particularly point out and distinctly claim the subject matter which applicant regards as the invention. 

5. The following is in regard to Claims 2 and 19. These claims recite the limitation "the second potential 
match set is based at least in part on a computation of reprojection error for matched pixels that resulted from a 
projective reconstruction of the second potential match set". According to this statement, the derivation of the 
second potential match set requires, at least in part, a projective reconstruction of the second potential match set. 
However, the projective reconstruction of the second potential match set assumes a priori knowledge of the second 
potential match set. The circuitousness of the above statement should thus be evident. In particular, as stated above, 
the determination of the second potential match set is based on a result that assumes this set has been determined a 
priori. The Applicant's specification does not adequately provide any insight as to what the Applicant intended to 
claim, so reference will be made to U.S. Patent Application Publication 2002/0 172413A1 (which corresponds to 
U.S. Patent Application Serial No. 09/825,266). Claims 2 and 19 of the this application seem to be directed toward 
the process depicted in Fig. 9 of U.S. Patent Application Publication 2002/0 17241 3 Al and discussed in paragraphs 
[0047]-[0056] of that publication. Seen in this light, a more appropriate description of the Applicant's claimed 
invention, as set forth in Claims 2 and 19, would entail the following: "the second potential match set is based at 
least in part on a computation of reprojection error for matched pixels that resulted from a projective reconstruction 
of the first potential match set", as opposed to the statement above. Claims 2 and 19 will be interpreted in this 
document as such. 

6. The following is in regard to Claim 3 and 20. Claims 3 and 20 suffer from the same ambiguity as in Claims 
2 and 19. These claims will be interpreted as though the "reprojection errors related to matched pixels in the first 
potential match set", as opposed to the "the second potential match set". 

7. The following is in regard to Claim 11. Claim 1 1 recites the limitation "at least one of the reference views". 
There is insufficient antecedent basis for this limitation in the claim. Claim 10, upon which Claim 1 1 depends, 
proposes only a single "reference view". It will be assumed, henceforth, that Claim 10 refers to at least one reference 
view, as opposed to a single reference view. 
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Rejections Under 35 U.S.C. 6 112(0 

8. The following is a quotation of the first paragraph of 35 U.S.C. 1 12: 

The specification shall contain a written description of the invention, and of the manner and process of making and using it, 
in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is 
most nearly connected, to make and use the same and shall set forth the best mode contemplated by the inventor of 
carrying out his invention. 

9. Claim 29 is rejected under 35 U.S.C. 1 12, first paragraph, as failing to comply with the enablement 
requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to 
enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the 
invention. 

10. The following is in regard to Claim 29. Though the Applicant does show the differentiation of E(g) with 
respect to parameter g, the Applicant fails to indicate anywhere in the specification that the differentiation E(g) is 
carried out using finite differences. (Incidentally, there is no mention of using finite differences in U.S. Patent 
Application Serial No. 09/825,266 either). 

Rejections Under 35 U.S.C. 8 102(b) 

1 1 . The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the 

rejections under this section made in this Office action: 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on sale in this 
country, more than one year prior to the date of application for patent in the United States. 

12. Claims 1, 4-9, 12-18, and 21-26 are rejected under 35 U.S.C. 102(b) as being anticipated by [Chen, 
VSMM99] (Chen and Medioni , "A Volumetric Stereo Matching Method: Application to Image-Based Modeling", 
IEEE 1999). 

13. The following is in regard to Claim 7. [Chen, VSMM99] describes a volumetric stereo matching method. 
The method involves the recovery of depth information - namely, the disparity surface d(u, v) (see [Chen, 
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VSMM99]: Abstract, sentence 1; page 29, right column, last paragraph 1 , sentences 1-3; and page 30, left column, 
paragraph 1, sentences 2-3) - for pixels of a base image representing a view of a scene. The method comprises at 
least the following steps: 

(l.a.) Detecting a plurality of pixels in a base image that represents a first view of a scene (e.g. 
capturing images from various viewpoints of the scene). It should be apparent from the 
various stereo images shown in [Chen, VSMM99] that the volumetric stereo matching 
method involves this step. Any one of these images can be arbitrarily denoted as a "base 
image". 

( 1 ,b.) Determining 3-D depth of the plurality of pixels in the base image by matching 

correspondence to a plurality of pixels in a plurality of images representing a plurality of 
views of the scene. 

Steps (l.a.)-(l.b) are typical of any stereo matching applications. The "matching correspondence" is discussed in 
Section 3.1 of [Chen, VSMM99]. The cross-correlation ®(u,v,d) is indicative of the matching correspondence 
between pixels of different images, each of which are representative of disparate views of the scene. 

(1 .c.) Tracing pixels (i.e. surface tracing) in a virtual piecewise continuous depth surface (i.e. 

disparity surface d(u.v)) by spatial propagation ([Chen, VSMM99] section 3.2, paragraph 
1, sentence 4) starting from the detected pixels in the base image (i.e. all pixels (h,v) of 
the base image) by using the matching and corresponding plurality of pixels in the 
plurality of images (e.g. seed pixels that satisfy a given cross-correlation criterion - 
[Chen, VSMM99] page 31, right column, paragraph 1, sentences 1-2) to create the virtual 
piecewise continuous depth surface viewed from the base image. Please refer to Section 
3 of [Chen, VSMM99]. 

(1 .d.) Each successfully traced pixel is associated with a depth in the scene viewed from the 
base image. This follows from the fact that traced pixels lie on the disparity surface. 



1 When referring to paragraphs in the cited references, the convention followed here is that the paragraph number is assigned to 
paragraphs of a given column (if applicable) or section, sequentially, beginning with the first full paragraph. Paragraphs that 
carry over to other columns will be referred to as the last paragraph of the column in which they began. 
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Again, the disparity surface is indicative of the depth of all visible surfaces of a given 

scene (page 29, right column, last paragraph, sentences 1-3). 
In this manner, the a volumetric stereo matching method of [Chen, VSMM99] conforms sufficiently to the method 
proposed by the Applicant in claim 1 . 

14. The following is in regard to Claim 18. Claim 18 recites essentially the same limitations as claim 1. (The 
3D reconstruction instructions merely implement the depth recovery method of Claim 1). Therefore, with regard to 
claim 18, remarks analogous to those presented above relative to claim 1 are applicable. 

15. The following is in regard to Claim 4. As shown above, [Chen, VSMM99] discloses a volumetric stereo 
matching method that adequately conforms to the depth recovery method of Claim 1. The volumetric stereo 
matching method of [Chen, VSMM99] further involves: 

Propagating a front of a virtual piece of a continuous depth surface (e.g. a surface front - 
this terminology is used in sentence 5 of the last paragraph of the right column on page 
3 1) to at least one neighboring pixel (e.g. each 4-neighbor of (w, v) - Fig. 4 of [Chen, 
VSMM99]) starting from the detected pixels (e.g. pixel (w, v)) in the base image. 
It should be apparent from the algorithm listed in Fig. 4 of [Chen, VSMM99] that the boundary (front) of the 
disparity surface is propagated. In this manner, the a volumetric stereo matching method of [Chen, VSMM99] 
conforms sufficiently to the method proposed by the Applicant in Claim 4. 

16. The following is in regard to Claim 21. Claim 21 recites essentially the same limitations as claim 4. (The 
3D reconstruction instructions merely implement the depth recovery method of Claim 4). Therefore, with regard to 
claim 21, remarks analogous to those presented above relative to claim 4 are applicable. 

17. The following is in regard to Claims 5-6. As shown above, [Chen, VSMM99] discloses a volumetric stereo 
matching method that adequately conforms to the depth recovery method of Claim 4. Tracing the disparity surface 
according to the volumetric stereo matching method of [Chen, VSMM99] further involves the following: 

(5. a.) Determining when a boundary is reached between two propagating fronts of virtual 
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pieces of a continuous depth surface ([Chen, VSMM99] page 31, right column, last 

paragraph, sentence 5). 
If the boundaries of two propagating fronts coincide then: 

(6.a.) Comparing the cross-correlations of the two propagating fronts about the reached 

boundary. This comparison is performed, for example, in the following step shown in 

[Chen, VSMM99] Fig. 4: 

disparity(w> f ) = <D(M>',^') > 0( u >',</")? d' : d" 

Clearly, a comparison of the cross-correlations, 0(« \v\d r ) and 0( u \ v \d "), is 

tantamount to a comparison of the matching costs of the two surface fronts. 
(6.b.) The surface front with the greatest cross-correlation prevails ([Chen, VSMM99] page 3 1 , 

right column, last paragraph, sentence 5). In other words, the propagation of the front 

with the lower cross-correlation is stopped. 
Step (6.b.) is equivalent to stopping the propagation of the front with the higher compared matching cost 2 . In this 
manner, the a volumetric stereo matching method of [Chen, VSMM99] conforms sufficiently to both the methods of 
Claim 5 and the method of Claim 6. 

18. The following is in regard to Claim 22-23. Claim 22-23 recite essentially the same limitations as claim 5-6, 
respectively. (The 3D reconstruction instructions of these claims merely implement the depth recovery methods of 
Claim 5-6). Therefore, with regard to claim 22-23, remarks analogous to those presented above relative to claim 5-6 
are respectively applicable. 

19. The following is in regard to Claim 7. As shown above, [Chen, VSMM99] discloses a volumetric stereo 
matching method that adequately conforms to the depth recovery method of Claim 1. Tracing the disparity surface 
according to the volumetric stereo matching method of [Chen, VSMM99] further involves the following: 

2 To see this recall how the Applicant defines matching cost (page 9, lines 1 0-1 5 of the Applicant's disclosure and, in ' 
particular, equation (8) on page 17). The matching cost is low for pixels that are likely matches and is high for unlikely 
matches. The relationship between the matching cost and cross-correlation (NCC) is shown in the Applicant's equation (8). 
Clearly, a high NCC results in a low matching cost £(g) and indicates a likely match, whereas a low NCC results in a high 
matching cost and indicates an unlikely match. Therefore, by stopping the propagation of the front with the lower NCC, one 
has effectively stopped the propagation of the front with the higher matching cost. 
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Propagating a front of a virtual piece of a continuous depth surface to at least one 
neighboring pixel surrounded by a predefined size window in the continuous depth 
surface. According to the volumetric stereo matching method of [Chen, VSMM99], the 
normalized cross-correlation (NCC) over a window (e.g. one with dimensions oxco and 
centered at pixels (u,v) and (u+d,v)) is used as the similarity measurement. See Section 
3.1 of [Chen, VSMM99]. 

In this manner, the a volumetric stereo matching method of [Chen, VSMM99] conforms sufficiently to the method 
of Claim 7. 

20. The following is in regard to Claim 24. Claim 24 recites essentially the same limitations as claim 7. (The 
3D reconstruction instructions merely implement the depth recovery method of Claim 7). Therefore, with regard to 
claim 24, remarks analogous to those presented above relative to claim 7 are applicable. 

21. The following is in regard to Claim 8-9 and 25-26. As shown above, [Chen, VSMM99] discloses a 
volumetric stereo matching method that adequately satisfies the limitations of Claims 7 and 24. The limitations set 
forth in Claims 8-9 and 25-26 were addressed previously in the discussion above relating to Claims 5-6 and 22-23. 
The details will not be repeated here. 

22. The following is in regard to Claim 12. [Chen, VSMM99] discloses a volumetric stereo matching method 
that includes the following: 

(12.a.) Providing a plurality of seed pixels that represent 3-D depth of the plurality of pixels in 
the base image view of a scene (i.e. seed voxels - [Chen, VSMM99] page 31, right 
column, paragraph 1). Seed voxels are determined by matching correspondence to a 
plurality of pixels in a plurality of images representing a plurality of views of the scene 
(i.e. ensuring that the cross-correlation 0(u,v, d) between the views satisfies 
0(w, v,d) > tl, tl being a predetermined threshold - [Chen, VSMM99] page 31, right 
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column, paragraph 1, lines 3-4). 
(12.b.) Tracing pixels in a virtual piecewise continuous depth surface by spatial propagation 
starting from the provided plurality of seed pixels in the base image by using the 
matching and corresponding plurality of pixels in the plurality of images to create the 
virtual piecewise continuous depth surface viewed from the base image, each 
successfully traced pixel being associated with a depth in the scene viewed from the base 
image. The surface tracing was treated above with respect to step (I.e.). See also [Chen, 
VSMM99] right column, paragraph 2 and Fig. 4. 

23. [Chen, VSMM99] presents the volumetric stereo matching method entirely as an algorithm and forgoes any 
mention of possible hardware implementations. However, the volumetric stereo matching method is shown to have 
been executed on a computer (e.g. SGI workstation - [Chen, VSMM99] page 34, left column, paragraph 2). In this 
case, item (12.a.) would be implemented in computer-readable instructions, say as a "image matching module", and 
item (12.b.) would be implemented in computer-readable instructions, say as a "propagation module". The "image 
matching module" and "propagation module" would presumably be resident (electrically coupled) in some memory 
which is, in turn, coupled to a central processing unit (CPU). In this manner, a computer implementation of the a 
volumetric stereo matching method of [Chen, VSMM99] would sufficiently conform to the image processing system 
of Claim 12. 

24. The following is in regard to Claim 13. As shown above, [Chen, VSMM99] suggests a computer 
implementation of a volumetric stereo matching method that adequately satisfies the limitations of Claims 12. The 
disclosed volumetric stereo matching method operates on photographic images (e.g. [Chen, VSMM99] Figs. 2 or 8). 
This necessitates some means of capturing images, such as a camera(s) (see [Chen, VSMM99] page 34, left column, 
paragraph 1). In order for the CPU to process the captured images, the camera would have to be somehow 
electrically coupled (e.g. via some camera interface) to the CPU. Therefore, the features of the image processing 
system proposed in Claim 13 are inherent to a computer-based implementation of the volumetric stereo matching 
method of [Chen, VSMM99]. 

25. The following is in regard to Claim 14. As shown above, [Chen, VSMM99] suggests a computer 
implementation of a volumetric stereo matching method that adequately satisfies the limitations of Claims 12. As 
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mentioned above the "image matching module" and "propagation module" are resident on a memory. Modern 
memory is implemented as an integrated circuit. All modern CPUs are implemented as integrated circuits. In this 
manner, a computer-based implementation of the a volumetric stereo matching method of [Chen, VSMM99] would 
adequately satisfy the limitations of Claim 14. 

26. The following is in regard to Claims 15-1 7. Claim 15-17 recite essentially the same limitations as claim 5- 
7, respectively. Therefore, with regard to claim 15-17, remarks analogous to those presented above relative to claim 
5-7 are respectively applicable. 

Rejections Under 35 U.S.C. § 103(a) 

27. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set 
forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if 
the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would 
have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

28. Claims 27-28 are rejected under 35 U.S.C. 103(a) as being unpatentable over [Chen, VSMM99]. 

29. The following is in regard to Claim 27. As shown above, [Chen, VSMM99] discloses a volumetric stereo 
matching method that adequately satisfies the limitations of Claims 24. Although [Chen, VSMM99] does not 
explicitly mention associating the normalized cross-correlation (NCC) with a matching cost, the usage of the NCC 
in [Chen, VSMM99] suggests that the two are closely related and could indeed be used interchangeably. As 
mentioned above, [Chen, VSMM99] uses the NCC, <3>(u,v f d), within windows associated with each of the 
propagating fronts (e.g. windows centered about the voxel (u,v,</) on the surface fronts - [Chen, VSMM99] page 31, 
left column, lines 1-3 and right column, paragraph 2, sentence 1) to determine which of two coincident surface 
fronts to continue to propagate ([Chen, VSMM99] page 31, right column, last paragraph, sentence 5). Furthermore, 
the NCC is indicative of the "correctness" of a match ([Chen, VSMM99], Section 3.2, paragraph 1, sentence 1). All 
of this is suggests the that the NCC provides an adequate measure of the matching cost of the propagating surface 
fronts. Therefore, it would have been obvious to one of ordinary skill in the art, at the time of the applicant's claimed 
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invention, to relate the matching cost of the propagating surface fronts to the computation of the NCC or designate 
the NCC as the matching cost. In either case, the motivation for doing so would have been to utilize the cross- 
correlative properties of the NCC to evaluate the degree to which voxels, comprising the surface fronts, represent 
correct matches. 

30. The following is in regard to Claim 28. As shown above, [Chen, VSMM99] discloses a volumetric stereo 
matching method that, when modified in the manner suggested above, adequately satisfies the limitations of Claims 
26. [Chen, VSMM99] suggests rectification of at least one pair of images corresponding to the base view of the 
scene and at least one of the other views (e.g. "reference views"). See [Chen, VSMM99] Section 2. Therefore, it 
would have been obvious to one of ordinary skill in the art, at the time of the applicant's claimed invention, to 
further perform rectification of at least one pair of images corresponding to the base view of the scene and at least 
one of the other views (e.g. "reference views"). The result of rectification is that "scanlines [epipolar lines] are 
parallel in each image, and the corresponding ones are collinear across the images" ([Chen, VSMM99] Section 2, 
paragraph 1, sentence 2 and last paragraph). This is known to have the effect of greatly simplifying the problem of 
finding correspondences across stereo images. 



31. Claims 2-3 and 19-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over [Chen, VSMM99], in 
view of [Zhang94] (Zhang, Deriche, Faugeras, and Luong, "A Robust Technique for Matching Two Uncalibrated 
Images Through the Recovery of Unknown Epipolar Geometry", INRIA 1994). 

32. The following is in regard to Claim 2. As shown above, [Chen, VSMM99] discloses a volumetric stereo 
matching method that sufficiently conforms to the depth recovery method proposed in Claim 1. The volumetric 
stereo matching method of [Chen, VSMM99] operates on multiple images (see, for instance [Chen, VSMM99] Fig. 
2 and Fig. 1 1). That is, the depth recovery according to this method includes the following: 

(2.a.) Detecting a plurality of image pixels in a first image corresponding to a first view of a 
scene. 

(2.b.) Detecting a plurality of image pixels in at least a second image corresponding to a 



Application/Control Number: 09/927,558 Page 12 

Art Unit: 2623 

respective at least a second view of the scene. The at least a second image deviates from 
the first image as a result of camera relative motion. This is typical of stereo matching 
algorithms (see also [Chen, VSMM99] page 34, Example 4). 
As noted above, the volumetric stereo matching method of [Chen, VSMM99], like most stereo matching methods, 
involves determining correspondences between a plurality of pixels in a first image and a plurality of pixels of at 
least a second image. In other words, the method of [Chen, VSMM99] further involves: 

(2.c.) Determining a first two-view correspondence between the plurality of detected image 
pixels in the first image and a plurality of detected image pixels in one of the at least a 
second image resulting in a first potential match set of candidate image pixels between 
the first image and the one of the at least a second image. This process involves the 
evaluation of the cross-correlation 0(w, v,d), as discussed above with regard to step 
(Lb.). 

33. [Chen, VSMM99] further suggests subsequently computing the projective reconstruction from the derived 
correspondence information in order to reconstruct the projective 3D structure of the observed scene ([Chen, 
VSMM99] Section 4, paragraph 1). 

34. [Chen, VSMM99] does not, however, expressly show or suggest: 

(2.d.) Determining a multiple-view correspondence between the plurality of detected image 

pixels in the first image and the plurality of detected image pixels in the at least a second 
image, the multiple-view correspondence being a refinement of the first two-view 
correspondence, resulting in a second potential match set of candidate image pixels 
between the first image and the at least a second image. 

(2.e.) The second potential match set is based at least in part on a computation of reprojection 
error for matched pixels that resulted from a projective reconstruction of the first 
potential match set. 

35. [Zhang94], on the other hand, discloses a stereo matching method. Stereo matching across disparate views 
of a scene is generally a fundamental step in depth recovery. The stereo matching method involves at least the 
following steps (refer to [Zhang94] Section 6.3 on pages 16-19): 
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(2.c.) Determining an initial set of correspondences or matches (e.g. a first two- view 

correspondence) between the plurality of detected image pixels in the first image and a 
plurality of detected image pixels in one of the at least a second image resulting in a first 
potential match set of candidate image pixels (e.g. candidate matches - see, for example, 
[Zhang94] page 10, Section 5, paragraph 1, lines 1-2) between the first image and the one 
of the at least a second image. 
(2.d.) Removing outliers from the initial set of matches (i.e. determining a "multiple-view 

correspondence" between the plurality of detected image pixels in the first image and the 
plurality of detected image pixels in the at least a second image). See, for example, 
[Zhang94] page 18, line 24 3 , in conjunction with the equations for the weights w, on 
pages 17-18. In this manner, the set of matches or correspondences thus obtained (e.g. 
the "multiple-view correspondences") represents a refinement of the "first two- view 
correspondence", resulting in a "second potential match set" of candidate image pixels 
between the first image and the at least a second image. 
(2.e.) The "second potential match set" is based at least in part on a computation of 

reprojection error (e.g. residual r t - [Zhang94] page 17, paragraphs 1-2) for matched 
pixels that resulted from "the difference between the z-th observation and its fitted 
value [4] " ([Zhang94] page 17, paragraph 1, lines 1-2). 
36. The teachings of [Chen, VSMM99] and [Zhang94] are combinable because they are analogous art. 
Specifically, both [Chen, VSMM99] and [Zhang94] disclose methods for stereo matching. Therefore, it would have 
been obvious to one of ordinary skill in the art, at the time of the applicant's claimed invention, to use the 
methodology of [Zhang94] - in particular, steps (2.d.)-(2.e.) above - to refine the initial set of correspondences 
derived according to the method of [Chen, VSMM99]. It would have been obvious to one of ordinary skill in the art, 



3 Lines are counted sequentially from the top beginning with 1 . An equation is treated as a single line, regardless of its length. 

4 It should be understood that the fitted value in this case is a projection of the matched pixels into some view being evaluated and the 
observation is the respective candidate point correspondences {(mij , m 2 i)}. See [Zhang94] page 17, last paragraph. 
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at the time of the applicant's claimed invention, that, in applying this refinement process to method of [Chen, 
VSMM99], the fitted values become the reprojected projective points that delimit the projective structure of the 
observed scene (i.e. "matched pixels that resulted from a projective reconstruction of the first potential match set"). 
The motivation for using the method of [Zhang94], in this manner, would have been to detect and remove outliers 
(e.g. bad locations and/or false matches - [Zhang94] page 16, last paragraph) from the initial set of candidate 
matches (e.g. those obtained via step (2.c.) above). See [Zhang94] Section 6.3. 

37. The following is in regard to Claims 19. Claim 19 recites essentially the same limitations as claim 2. (The 
3D reconstruction instructions merely implement the depth recovery method of Claim 2). Therefore, with regard to 
claim 19, remarks analogous to those presented above relative to claim 2 are applicable. 

38. The following is in regard to Claim 3. As shown above the methods of [Chen, VSMM99] and [Zhang94] 
can be combined to satisfy the limitations of Claim 2. [Zhang94] further disclose that the "outlier-less" set of 
correspondences (i.e. the "second potential match set") is based at least in part on a least median of squares 
computation (LMedS ~ [Zhang94] Abstract and page 17, lines 4-7 and paragraph 2) of the reprojection errors (i.e. 
residuals r, - see above) related to matched pixels in the first potential match set. LMedS, as put forth by [Zhang94], 
has an advantage over other methods in that it "is very robust to false matches as well as outliers due to bad 
localization" ([Zhang94] page 17, paragraph 1, last sentence). Therefore, it would have been obvious to one of 
ordinary skill in the art, at the time of the applicant's claimed invention, to use the LMedS of [Zhang94] to detect 
and remove outliers from the first potential match set. 

39. The following is in regard to Claim 20. Claim 20 recites essentially the same limitations as claim 3. (The 
3D reconstruction instructions merely implement the depth recovery method of Claim 3). Therefore, with regard to 
claim 20, remarks analogous to those presented above relative to claim 3 are applicable. 

40. Claims 10-1 1 and are rejected under 35 U.S.C. 103(a) as being unpatentable over [Chen, VSMM99], in 
view of [Okutomi93] (Okutomi and Kanade "A Multiple-Baseline Stereo", IEEE 1993), in further view of 
[Lewis95] (Lewis, "Fast Normalized Cross-Correlation", 1995). 
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41. The following is in regard to Claim 10. As shown above, [Chen, VSMM99] discloses a volumetric stereo 
matching method that sufficiently conforms to the depth recovery method proposed in Claim 9. As discussed above, 
the method of [Chen, VSMM99] involves the computation of the NCC between a first image window of a pre- 
determined size in the base view, and a second image window of the same pre-determined size in other views 
([Chen, VSMM99] Section 3.1). It was also show above that the NCC is intimately related to the matching cost. 
Despite this, [Chen, VSMM99] does not expressly show or suggest that the matching cost is determined by 
computing the summation of all the normalized cross-correlations between a first image window of a pre-determined 
size in the base view, and a second image window of the same pre-determined size in a reference view. 

42. [Okutomi93] disclose a stereo matching method that uses multiple stereo pairs with various baselines for 
the purposes of depth recovery ([Okutomi93] Abstract sentence 1). According to [Okutomi93], the summation of the 
sum of squared differences (SSD) from multiple stereo pairs can be used to indicate the "correctness" of a set of 
matching points (pixels). See [Okutomi93] page 353 (right column, paragraph 3, first and last sentence), page 354, 
(left column, lines 7-10), page 355 (Section B, paragraph 2, sentence 1), and equation (39) on page 362. This 
summation is analogous to a matching cost in that it approaches a minimum for correct matches ([Okutomi93] page 
353, right column, paragraph 3, last sentence). According to equation (39) the computation is made within a 
predetermined window W belonging to each of the stereo images ([Okutomi93] page 354, right column, paragraph 
1, line 4). In summary, [Okutomi93] shows a "matching cost" determined by computing the summation of all the 
SSD 's between a first image window of a pre-determined size in the base view and a second image window of the 
same pre-determined size in other views (e.g. "reference views"). Given this, it would have been obvious to one of 
ordinary skill in the art, at the time of the applicant's claimed invention, to use a "matching cost" (i.e. a measure of 
the similarity or "correctness" of a match) determined by computing the summation of all the SSD 's between a first 
image window of a pre-determined size in the base view and a second image window of the same pre-determined 
size in other views (e.g. "reference views"). As pointed out by [Okutomi93], this would have the effect of 
advantageously reducing global mismatches ([Okutomi93], page 353, right column, paragraph 3, first sentence). 

43. Note that the SSD and NCC are closely related mathematically. As measures of similarity among regions of 
images, they are functionally equivalent. See [Lewis95] Section 2, paragraph 1. Furthermore, it can also be shown 
that the process of minimizing the SSD and maximizing the NCC are in fact equivalent operations. All of this 
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suggests that SSD and NCC are essentially interchangeable. Moreover, the NCC has the advantage over the SSD in 
that it is invariant to "changes in image amplitude such as those caused by changing lighting conditions across the 
image sequence" ([Lewis95] page 2, lines 8-17). Since lighting conditions are likely to vary across the various stereo 
images, it would have been obvious to one of ordinary skill in the art, at the time of the applicant's claimed 
invention, to substitute the NCC for the SSD in the stereo matching method of [Okutomi93]. This would yield a 
"matching cost" determined by computing the summation of all the NCC's between a first image window of a pre- 
determined size in the base view and a second image window of the same pre-determined size in other views (e.g. 
"reference views"). 

44. The teachings of [Chen, VSMM99] and [Okutomi93] are combinable because they are analogous art, 
namely, the art of stereo matching. The teachings of [Lewis95] are applicable to both the methods of [Chen, 
VSMM99] and [Okutomi93] because all relate generally to correlative approaches for evaluating the similarity of 
windowed-regions belonging to a collection of images of a scene. Combining these teachings as suggested above 
yields a method that conforms sufficiently to that of claim 10. 



45. The following is in regard to Claim 11. As shown above the teachings of [Chen, VSMM99], [Okutomi93], 
and [Lewis95] can be combined so a to yield a method that adequately satisfies the limitations of Claim 10. [Chen, 
VSMM99] suggests rectification of at least one pair of images corresponding to the base view of the scene and at 
least one of the other views (e.g. "reference views"). See [Chen, VSMM99] Section 2. Therefore, it would have 
been obvious to one of ordinary skill in the art, at the time of the applicant's claimed invention, to further perform 
rectification of at least one pair of images corresponding to the base view of the scene and at least one of the other 
views (e.g. "reference views"). The result of rectification is that "scanlines [epipolar lines] are parallel in each 
image, and the corresponding ones are collinear across the images" ([Chen, VSMM99] Section 2, paragraph 1, 
sentence 2 and last paragraph). This is known to have the effect of greatly simplifying the problem of finding 
correspondences across stereo images. 



V 
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Allowable Subject Matter 

Objections, Allowable Subject Matter 

46. Claim 29 would be allowable if rewritten or amended to overcome the rejection(s) under 35 U.S.C. 1 12, 
first paragraph, set forth in this Office action. 

47. The following is a statement of reasons for the indication of allowable subject matter. As shown above, 
[Chen, VSMM99] discloses: 

(29.a.) Tracing at least one parameter surface (e.g. disparity surface), each of the at least one 
parameter surface traced starting from at least one predetermined seed pixel point. 
However, no prior art methods were encountered that involved or even suggested the step of: 

(29.b.) Calculating a derivative of function E(g) with respect to parameter g by using finite 
difference to minimize the equation of E(g) put forth in Claim 29. 



Citation of Relevant Prior Art 

48. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: 
[ 1 ] Szeliski, Stereo Algorithms and Representations for Image-Based Rendering. 1 0 th British 
Vision Conference. 1999. 

Szeliski discusses various volumetric depth map representations, as well as various 
approaches to solving the stereo correspondence problem through global optimization. 
[2] U.S. Patent 5,91 7,937. Szeliski et al. . Publication Date: Jeune 1999 

Szeliski et al. propose a volumetric approach to deriving the disparity surface. In particular, 
the method involves an evaluation of the reprojection error. 
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[3] Ohta and Kanade, Stereo by Intra- and Inter-Scanline Search Using Dynamic Programming. 
IEEE. 1985. 

Ohta and Kanade cast the stereo correspondence problem as that of finding in a 3D search 
space an optimal matching surface that satisfies inter- and inter-scanline coherency. The 
method includes the evaluation of a cost of the matching surface which is defined as the sum 
of the costs of the intra-scanline matches on the 2D search planes. 
[4]-[7] propose a maximum-flow formulation of the N-camera stereo correspondence problem. Similar to the 
methods above, [4]-[7] is essentially a volumetric realization of the stereo correspondence problem. The method of 
[4]-[7] share several common features with the Applicant's claimed depth recovery methodology. 
[4] U.S. Patent 6,046, 763. Roy. Publication Date: April 2000. 

[5] Roy and Cox, A Maximum-Flow Formulation of the N-Camera Correspondence Problem. 
IEEE 1998. 

[6] Roy, Stereo Without Epipolar Lines: A Maximum-Flow Formulation. International Journal of 

Computer Vision 1998. 
[7] Zhao, Global Optimal Surface from Stereo. IEEE 2000. 
49. [8]-[12] discuss in greater detail the method of [Chen, VSMM99]. 

[8] Chen and Medioni. Efficient Interative Solution to M- View Projective Reconstruction. IEEE 
1999. 

[9] Chen and Medioni. A Semi-automatic System to Infer Complex 3-D Shapes from 

Photographs. IEEE 1999. 
[10] Chen and Medioni. Building Human Face Models from Two Images. Proceedings IEEE 

Workshop on Multimedia Signal Processing 1998. 
[11] Chen, Multi- View Image-Based Rendering and Modeling. Ph.D. Thesis, University of 

Southern California, May 2000. 
[ 1 2] Chen, Robust Point Feature Matching in Projective Space. IEEE 200 1 . 
The following are other, newer volumetric approaches to deriving the disparity surface 
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[13] Sun, Fast Stereo Matching Using Rectangular Subregioning and 3D Maximum-Surface 

Techniques. International Journal of Computer Vision. 2002. 
[14] Tang, Tsui and Wu. Dense Stereo Matching Based on Propagation with a Voronoi Diagram. 

Indian Conference on Computer Vision, Graphics, and Image Processing. 2002 
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